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ABSTRACT 

Applications of multi-sensor fusion span a variety of domains from crisis management, military tactical 
situation and treat assessment, environmental monitoring, and more recently, monitoring of information 
systems. Rapid advances in data collection and dissemination provide the opportunity for major 
improvements in the information gathering aspect. However, a fundamental paradox exists in the 
understanding side. The paradox is that information analysts are drowning in a sea of data but unable to 
obtain the knowledge that they need to address difficult problems. This has often been referred to as the 
data overload dilemma or more recently framed "cogmenutia fragmentosa , \ On one hand, an 
unprecedented capability exists to collect data via distributed sensors, commercial information providers, 
human sources, or Internet resources. Smart micro-scale sensors, wireless communications, and global 
Internet accessible resources enable the entire earth to be a potential information resource. Such 
information is available literally at the fingertips of the analysts. However, the wealth of data has not 
produced a commensurate improvement in analyst abilities. Analysts are literally swamped with data. 
They have a wide variety of choices to make as to what is useful and usable, given the context of what they 
are trying to understand. 

This paper describes a three-pronged approach to improve information understanding including; (1) use 
of 3-D visualization and interaction techniques, (2) role-playing gaming (RPG) concepts, and (3) use of 
team-based intelligent advisory agents (cyber-advisors). The environment promotes rapid development 
and evaluation of hypotheses regarding evolving complex situations in an environment in which enormous 
amounts of data and information are available, but for which there is no clear mapping between 
observables and underlying threat conditions or activities. Use of advanced visualization techniques and 
gaming concepts assist in focusing the analysts ’ attention and promotes an interactive, creative analysis 
process in which hypotheses are formulated, evaluated, criticized, modified, and changed. The use of 
gaming techniques leverages the skills of new analysts, already experienced in gaming technologies. 

Comments are made concerning the application of this approach to understanding network information. 
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A Three Pronged Approach for Improved Data Understanding: 

3-D Visualization, Use of Gaming Techniques, and Intelligent Advisory Agents 



1.0 INTRODUCTION 

At the Pennsylvania State University, we are developing techniques to enable analysts to perform their 
jobs faster and more accurately as both individuals and as members of an analyst team. Our concept, 
illustrated in Figure 1, involves three major innovations: 

(1) A 3-dimension, full-immersion environment with new displays and multi-sensory interaction, using 
concepts such as deliberate synesthesia, will enhance the ability for analysts to understand an 
evolving situation and to extract patterns in complex, large data sets; 

(2) An engaging gaming environment, modeled after role-playing games (RPGs), will allow analysts to 
alternate between a strategy-based game interface (a macro decision-maker who plays a “god-like” 
role to assimilate data and develop hypotheses) and an RPG interface (a micro decision-maker who 
interacts with characters who will tell him/her stories which assist in the quest to determine the 
accuracy of evolving hypotheses and to create an interpretation of a situation or threat); and 

(3) A virtual cyber-advisory team will provide expert advice in specialized domains, seek fallacies in 
evolving hypotheses or interpretations, bring forward historical case information, and occasionally 
act as “curmudgeons” who seek to question the arguments developed by an analyst. The advisory 
team will be modeled using team-based intelligent agents based on the recognition-primed decision 
(RPD) cognitive model. 


These concepts are being developed and tested in a living laboratory environment to determine the 
effectiveness of the tools and cognitive aids. 
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Figure 1: Concept of an analysis environment allowing analysts to use 
advanced 3-D, RPG gaming environment and a cyber-support team 
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2.0 MOTIVATION AND PROBLEM FOCUS 

The specific problem we are addressing involves how to address the increasing “glut” of data provided to 
analysts, resulting in analysts “thirsting for knowledge” while “drowning in a sea of data”. In particular, 
we seek visualization techniques, environments, and cognitive aids to cross the longest yard - 
transforming data from a network centric collection/fusion system (as data rapidly accrues in an evolving 
situation data base into actionable knowledge inside an analyst’s head (Figure 2)). 
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developed to cross the 
longest yard - 
transforming information 
in a computer system into 
actionable knowledge for 
analysts 


Energy Signals Data State vectors Labels Knowledge 


Figure 2: A key challenge is transforming from data inside an evolving situational data 
base to actionable knowledge for decision-making and understanding 

2.1 Barriers to solve the problem 

A fundamental paradox exists in information fusion. Information fusion in this context may be used in 
traditional areas such as national defense, counter-intelligence or situation assessment for tactical military 
applications (Hall and Llinas, 2001). Alternatively, information fusion may involve emerging applications 
such as emergency crisis management, environmental monitoring, or monitoring complex systems such as 
machines or industrial processes. The paradox is that information analysts are drowning in a sea of data 
but unable to obtain the knowledge that they need to address difficult problems. This has often be referred 
to as the data overload dilemma (Kuperman, 2001) or more recently framed "cogmenutia fragmentosa" 
(McNeese & Vidulich, 2002). 

On one hand, an unprecedented capability exists to collect data via distributed sensors, commercial 
information providers (e.g., AccuWeather, Library Services, and commercial search businesses), human 
sources, or Internet resources. Smart micro-scale sensors, wireless communications, and global Internet 
accessible resources enable the entire earth to be a potential information resource. Such information is 
available literally at the fingertips of the analysts. In particular, the Internet has exceeded one billion web 
pages, with a continuing exponential increase. However, the wealth of data has not produced a 
commensurate improvement in analyst abilities. Analysts are literally swamped with data. They have a 
wide variety of choices to make as to what is useful and usable, given the context of what they are trying 
to understand (Woods, 1998). 
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On the other hand, the glut of data can be overwhelming and may inadvertently promote poor decision 
processes (Ferran, 1999). Studies of decision-making under stress have shown that too much information 
can cause ineffective decision styles. An example is the hyper-vigilance mode , in which a decision-maker 
frantically searches for new information, without taking time for reflection and thoughtful analysis of 
existing data. The huge glut of rapidly changing data via the Internet may encourage this type of response. 
Alternatively, a decision-maker may feel overwhelmed with new information and simply ignore new data. 
Thus, in a rich atmosphere of data, decision-makers are suffocating for knowledge (McNeese and 
Vidulich, 2002). They may have a large amount of cognitive readiness available to fuse multiple 
information sources but in fact their meta-cognition (McNeese, 2000) may be very limited. This often 
makes decisions about ’’what to do next” daunting. 

2.2 Overcoming the barriers 

We are addressing these barriers by an approach that; (a) provides new methods for interacting with data 
and meta-data using multiple human senses, (b) involves a gaming environment to capture the imagination 
of analysts and make the data analysis process engaging and fun, and (c) provides cognitive aids using 
team-based intelligent agents to guide the analysis process and critique evolving hypotheses. The intent is 
to treat analyst attention as a limited and critical resource that must be managed and conserved for 
improved analysis. We will develop cognitive models (using a knowledge elicitation approach on 
experienced analysts), implement software prototypes to demonstrate the concepts, and evaluate the 
effectiveness of the tools and techniques using human subjects in a living laboratory environment. 

2.3 Current Applications 

For crisis management and related applications, we have developed cognitive models (using a knowledge 
elicitation approach on experienced analysts), implemented software prototypes to demonstrate the 
concepts, and evaluated the effectiveness of the tools and techniques using human subjects in a living 
laboratory environment. The living laboratory environment links observations of analysts working in real 
environments, to development of cognitive models using a formal knowledge elicitation approach, and 
development and evaluation of prototype tools. The tools are tested in a simulation environment using 
teams of human subjects to quantitatively evaluate the effectiveness of the tools. A special laboratory 
environment and simulation tool (neoCITIES) is used for the evaluation. The implementation of these 
tools leverages several special laboratory facilities including: a synthetic environment applications 
laboratory containing a full immersion, 3-dimensional visualization facility, a laboratory for intelligent 
agents, and the user science and engineering laboratory. In addition gaming tools and visualization 
creation tools are used from our real-time aesthetics and experience laboratory. 
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3.0 THEORIES AND TECHNIQUES GUIDING THIS RESEARCH 

3.1 New 3-D Multi-sensory Techniques for Data Interaction 

Innovative concepts are being developed and demonstrated for 
improved information retrieval, data understanding and 
interpretation (McNeese and Hall (2003). A new concept of 
intelligent information interpretation, search and retrieval 
(I 3 SR) is being developed by exploring concepts such as multi- 
sensory interaction, dynamic computer-guided focus of 
attention, deliberate synesthesia, utilization of negative space 
concepts, and adversarial game concepts. This effort uses The 
Pennsylvania State University Synthetic Environment 
Applications (SEA) laboratory to explore the use of new 
multi-sensory interactions: e.g., sound, vision, haptic 

interfaces. Demonstrations are being developed to explore the 
concepts identified above and also exploring user/information 
control and feedback mechanisms. Examples of potential 
demonstrations include novel, multi-sensor interactions for 
data display and interaction; utilization of deliberate 
synesthesia effects, use of negative space and blink 
comparison concepts, deliberate blurring and transparency of 
displays and novel fly by electronic-wire control concepts. 

Figure 3 shows an example of a situation display in which height above the floor represents time of year, 
different shapes represent different data types, and sound is used to indicate “goodness of fit” (viz., 
measure of association) among different data types. 

3.2 Game-based Techniques for Improved Visualization and Analysis 

We are exploring several directions, based on video gaming, to create tools that would enable analysts to 
perform their jobs faster and better. Our goal is to explore the utility of video gaming (a) as visualization 
methods and (b) as interaction models to: 

1. Increase productivity of analysts by a) providing an engaging interaction method based on game 
design patterns and b) providing a visualization method based on game methods. This is based on 
the belief that next generation analysts are game players and thus are familiar with the 
visualization methods used in games. Providing a closer visualization method to what analysts are 
accustomed to will enable them to assimilate information quicker. Game visualization methods 
have been successfully used in mainstream games to enable quick decision making in a game type 
environment, where making decisions quickly is an important element of game play. 

2. Increase the speed by which analysts extract information by providing an abstract visualization 
method based on cinematic/theatric techniques that allow users or audience members to grasp the 
story quickly through the set design, character composition and music. 

3. Enhance the quality of hypotheses generated by a) providing a method for visualizing hypotheses 
to assist in revealing contradictions or visualizing probable hypotheses and b) providing a gaming 
technique for finding contradictions and holes in hypotheses or stories 
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We are building a 3-D environment where analysts alternate between a strategy-based game interface 
(macro decision-maker) and a Role Playing Game interface (micro decision-maker). We believe 
visualization and interaction methods provided through RPG (Role Playing Games) and Strategy-based 
games can provide an environment that achieves the goals set above. 


An example screenshot from a strategy-based game is shown in Figure 4. In a typical strategy-based game 
(e.g., Warcraft , Civilization ), the player is omniscient. He/she looks at the world from above. Interaction 
methods mainly involve strategic actions concerning resources, such as allocation of soldiers and directing 




Figure 4: Screen shot from 
Civilization IV 


action of specific units or groups of units. Several 
visualization techniques are used in these types of 
games. For example, the play often observes the 
world as a 2-D or 3-D map. Players can zoom in and 
out of the map or can maneuver in the world space to 
different parts of the map. There are several symbols 
that are used on the map to indicate specific 
information, such as names of region or resources. In 
Figure 4, boundaries of specific countries or 
civilizations are shown in the map. Also, there are 
context-based icons or menus that are usually 
displayed based on the user cursor or upon execution 
of specific events. 


We are also exploring several directions in mapping a strategy-based game interface within the analyst 
domain and measure its utility in enhancing analyst productivity, speed, and quality of hypotheses 
generated. Several explored visualization techniques for analysts have assumed a removed point of view 
with a GIS (Geographic Information System) (Risch, Rex et al. 1997; Rex 2002; Chen, Atabakhsh et al. 
2005; Corporation 2005). These works are similar to the visualization methods used in strategy-based 
game. Strategy-based game interfaces have evolved over many years. The interfaces have evolved 
adopting a very minimal context-sensitive interface to visualize dynamic and static information and 
events. We develop several mappings between analyst’s data and the current types of displays or 
visualizations used by strategy based games. We envision using graphics elements, such as maps, bars, 
sprites representing people, to represent different elements such as events, people, and their relationships, 
etc. Our goal is to use gaming techniques as well as cinematic techniques to visually depict analyst related 
data in a way that allows analysts to quickly grasp the situation and the event. 


Since the strategy-based game interface provides a removed interface, analysts may decide to dig deeper 
into a particular accumulated data abstracted in the strategy-based interface. In such cases, analysts will 
need another type of interface. We will evaluate the utility of an RPG-based game interaction model and 
visualization method. 



Figure 5: Screenshot from Neverwinter Nights 
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Screenshots from the RPG game, Neverwinter Nights are shown in Figure 5. In RPG games, a player 
assumes the role of a character within the game. He/she normally experiences the narrative from a third or 
first person interface as opposed to the God view as in strategy-based games. In these games the 
interaction model involves talking to different characters, solving quests, or fighting enemies. In role- 
playing games the player often confronts many characters who will all tell him/her their stories. Some of 
these stories are relevant to the game mission and others serve as side missions or quests that players take 
on if they desire. 

We are also developing a 3D representation of the current reports that analysts analyze. In this 3D 
representation, the analysts will be immersed in a 3D world similar to the ones used in RPG games. The 
analyst will be able to talk with characters who give him/her specific information, e.g. CNN reporter 
exposing a specific story that was on the CNN website, etc. As the characters talk, we envision using 3D 
abstract graphics to show the story, using similar visual metaphors as used in the movie Mirror Mask 
(http://www.sonypictures.com/homevideo/mirromask/). To compose such visual representations of stories, 
we will develop an artificial intelligence (AI) system that maps a story instance to an abstract story 
segment based on a composition of story patterns that we will develop based on example stories that 
analysts monitor. These story patterns will be developed following the same methods used by Propp 
(Propp, Wagner et al. 1968). Using these story patterns and the instance story, the AI system will compose 
a 3-D abstract representation. 

Finally, we are exploring the utility of gaming in hypothesis visualization and evaluation. We will also 
explore the potential of using games as a storyboarding technique to visualize a hypothesis and evaluate its 
validity. The specific goal of this task is to increase analyst engagement and the quality of the hypothesis 
he/she generates. 

Many analysts engage in a process of hypothesis generation, composition, and rating. Currently this 
process all happens in the analyst’s head. As a result, analysts can only generate simple hypotheses that 
are formulated based on only a small amount of information that can reside in one person’s head at any 
moment in time. These hypotheses may then be invalid or less complicated than what they should be. We 
will enhance this process by exploring a storyboarding game-based technique. We aim to give the analyst 
a way to sketch out their stories or hypotheses using a SIMs like environment. 



Figure 6: Scenarios Created using The Sims 2 


The Sims 2 (the second version of The Sims game) is a game that is similar to a doll house in 3-D (Figure 
6). Players can make their own custom character using a base set of characters. They can then customize 
their personalities and construct houses for them and put them in these houses. They can then direct the 
characters and initiate scenarios to tell a story, e.g. make two characters fall in love or kill another 
character. The Sims was a major hit and many people worked for hours putting together their own 
scenarios and characters. Will Wright (the creator of the Sims ) reported receiving over 75,000 customized 
characters for The Sims game. 
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We believe using the same motif for a doll house would provide an engaging method for analysts to 
storyboard hypotheses. In addition, this method would also allow them to visualize the hypothesis and 
may lead to composition of complex scenarios and early discovery of conflicts and different scenarios or 
hypotheses. 

The Sims as is, however, will not suffice; it would be too low-level for an analyst-type hypothesis and may 
take a long time to construct a relevant scenario. We believe the analyst needs a tool that will allow 
him/her to quickly put together a scenario and play with it. For this purpose, we will build a Sims-like 
environment with add-on tools that allow analysts to drag and drop story patterns from a set of defined 
story patterns that we will compose based on Propp’s morphology (Propp, Wagner et al. 1968). We will 
also build several character models that represent several stereotypes that analysts can use to depict their 
scenario. The use of stereotypes and story patterns will enable analysts to quickly visualize the hypotheses 
and identify problems with them. The system will include several rules that enable some low-level AI, 
allowing characters to react to events represented in the story patterns, and thus creating a world dynamics 
based on the patterns dropped in the world, similar to The Sims. 

Once hypotheses are collected through this tool, the game will begin. Analysts can invoke a debate where 
other analyst teams (composed of virtual and real analysts) try to find contradictions to the hypotheses 
presented. Analysts would use the tool to drag and drop patterns and maneuver characters, thus showing 
the contradictions or other possible hypotheses. 

3.3 Cyber Advisory Team 

A framework is currently being developed to use intelligent software agents to assist in data understanding 
and situation assessment. The framework is based on a team-based agent environment developed by Yen 
(Yen et al (2004). In this model, intelligent agents act similar to the way a good human team operates; 
namely cooperating in a dynamic and positive way, sharing a team mental model of the decision and 
analysis process, and proactively sharing information among team members to improve their analysis and 
decision-making performance. The framework developed by Yen includes an architecture, knowledge 
representation and reasoning methods and internal information exchange language to emulate human 
teams. In addition, the architecture is based on the recognition primed decision (RPD) model of human 
teams in complex, dynamic environments. 

The intelligent agent concept will allow 
creation of a virtual advisory team to 
support the analysis process (Figure 6). 

The concept is analogous to a defense or 
prosecution team that uses a team of 
special experts, specialists to search for 
related case studies, analysts to watch jury 
reactions and others to assist in 
dynamically preparing and modifying a 
defense or prosecution. By analogy, 
analysts are developing an understanding 
of an evolving situation and creating and 
assessing hypotheses regarding the case - 
modifying their approach as more data 

are uncovered or new interpretations are Fjgure 6; | ntemg ent Agents as a Virtual Advisory 

brought forth. One or more intelligent Team 

agents could act as “curmudgeon” 

agents, designed to guard against known cognitive biases such as the confirmation bias (in which a human 
seeks information that only confirms his or her hypothesis rather than looking for refuting evidence). 
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4.0 METRICS AND MEANS OF EVALUATION 

Numerous tools and techniques are being developed or adapted to assist the data analysis process. 
While proponents argue that their proposed method or tool will “significantly improve” the analysis 
process, such claims are difficult to validate without implementing the proposed tool in an operational 
environment. This is problematic for many reasons including disruption of already overloaded 
operational centers and analysts need to train analysts on the utilization of the new tool or technique, and 
difficulty in making quantitative comparisons of the analysis process “with” and “without” the introduced 
tool. Moreover, even if the proposed tool or technique is implemented it becomes a “point solution” and 
does not allow experiments to determine how variations of the tool or technique would affect the analysis 
performance. Our research uses the concept of a “living laboratory” (Figure 7). The concept involves a 
simulation environment (NeoCITIES) that allows introduction of tools such as collaboration aids or 
cognitive tools into a team decision environment using human test subjects. The subjects act in a team 
decision environment and are given the task of analyzing an evolving situation and making decisions. 
The simulation environment (via NeoCITIES) provides test subjects with reports and information related 
to an evolving scenario. The subjects work in teams (with separate roles for each team member) to assess 
the situation. The simulator drives the analysis process and collects data on the efficacy and accuracy of 
the student teams. In this way statistical information can be collected for quantitative evaluation of 
introduced tools. 


A unique living laboratory 
environment provides the 
ability to support the 
design of effective analyst 
support tools and to 
quantify their utility 
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Figure 7: Concept of a Living Laboratory for Quantitative Evaluation of Tools 
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5.0 APPLICATION TO NETWORK VISUALIZATION AND ANALYSIS 


The on-going research described 
in this paper can also be applied 
to the visualization and analysis 
of network systems. For 

example, figure 8 shows a concept 
for visualizing a computer 
network in which local clusters of 
computers interact with each other 
and interact with a wider set of 
computers and “networks of 
networks” via the internet. 

Visualization tools, intelligent 
agents, and gaming techniques 
can be used for monitoring the 
health of the network (e.g., 
utilization of communications, 
overload of communications 
channels, undue use by specific 
individuals, etc). An common 
example of such overload occurs Fi 9 ure 8: Example of Visualizing a Network 

at Pennsylvania State University at the start of each fall semester, when numerous new students arrive on 
campus and begin downloading extensive amounts of audio and video files. 
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We believe these general research thrusts will assist in understanding and managing the huge amounts of 
data to be collected and fused for both traditional and emerging applications. 
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Michael McNeese 
John Yen 
Magy Seif El-Nasr 
The Pennsylvania State University 
College of Information Sciences and Technology 


Prepared for the NATO Workshop on Visualizing Network Information 
held in Copenhagen, Denmark, 17-20 October, 2006 





PhNN State 


College of 

Information Sciences and Technology 


Presentation Outline 


• A Brief Advertisement 

• The Data Understanding Challenge 

• A Three-Pronged Approach 

- Visualization 

- Intelligent Agents 

- Gaming Concepts 

• Application to Network Visualization 
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1ST Class of ’06 Heads to the Work Place 


It looks to be an "up" year for the Class of 2006: average 
starting salaries and signing bonuses have increased 
over last year’s; the number of companies looking to hire 
1ST students continues to grow; and the market segments 
of those companies is expanding. 

"Everything is positive," said Shaun Knight, IST’s 
professional placement coordinator, of the job prospects 
oflST’s May graduates. 

About 75 percent of 1ST seniors have found full-time 
placement or have opted for graduate school, said Knight, 
basing his estimate on recent student-survey data and 
anecdotal evidence. While similar to rates from previous 
year's, the Class of 2006 is the largest ever graduating 
class. 

Salaries are averaging $56,1 68, up three percent from 
2005, with signing bonuses jumping 50 percent from last 
year to an average of $5,250, Knight said. 

"Demand for 1ST students is high, and we are seeing 
interest in our graduates from a wide range of companies 
including those in manufacturing, financial services and 
defense," Knight said. 

Students are being hired as information systems 
engineers, security architecture analysts and certification 
and accreditation consultants as well as joining 
leadership development programs. Among the new 
companies hiring 1ST students are Bank of America, 



Coming to You Soon 


About the College of Information Sciences and 
Technology 

1ST was founded to build leaders for the digital, global 
economy. From the beginning, corporate experts have 
advised us to build a program that helps graduates 
become technologically skilled but also able to 
understand how technology can make a difference in 
business, government, social services, education, and 
many other fields. The core values of 1ST - respect and 
diversity - drive our culture and educational methods and 
help to make the 1ST program unique. The program is 
highly interdisciplinary, and special-topics courses are 
created as the business landscape changes so that 
students get real-time information about the IT world. 
Additions and improvements to 1ST courses are regularly 
made based on feedback from student control groups. 

Follow the link below to learn firsthand what 
, the 1ST student experience is all about from 
' graduate Steve Pipino (1ST ’03). Now an IT 
consultant with IBM, Steve talks about how 
the college prepared him for his career and more. (This 
video was originally recorded for an 1ST event held April 
15, 2005.) 

Steve Pipino Video (Requires Windows Media Player) 


Progress Made on New Security and Risk Analysis 
Major 

Penn State's proposed new major—the B.S. in Security 
and Risk Analysis (SRA) to be offered through 
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In This Section 

Dean Search in Progress 

Dean's Picks 

Meet the Dean 

D? IST2U 

1ST At Penn State’s Campuses 

Our Vision, Mission, and 

Values 

Meet Our Faculty 

The 1ST Advisory Board 

Giving 

The 1ST Building 

In This Section 

Prospective Students 

Current Students 

Faculty & Research 

Partners 

1ST 

News and Events 

Stay Informed 


1ST Features 


Strategic Plan 

-i 


http://ist.psu.edu 

College of Information 
Sciences and Technology 

• Interdisciplinary program links 
computer science, users, and 
information technology 

• Founded in 1998 

• 175 faculty members 

• 2600 undergraduate students 

• 20 campus locations 

• Associate, BS, MS, PhD degrees 

• 85 graduate students 

• 3000 interns 

• > $ 30 M research funding 

• 70 + new courses 

• On-line courses & distance 
education 


The Penn State College of Information Sciences and Technology is an interdisciplinary 
program that integrates computer science, information technology and system users at the 
individual, group, organizational and societal levels. Degrees are offered at the associate, 
baccalaureate, masters, and PhD levels. 
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A key challenge: Crossing the longest yard 
to transform energy into knowledge 


CONTEXTUAL INTERPRETATION 







Earth/World 
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The longest yard 



HUMAN-CENTERED 
ADAPTIVE 
MULTI-SOURCE 


Models & Syntax 


Advanced tools will be 
developed to cross the 
longest yard - 
transforming information 
in a computer system into 
actionable knowledge for 
analysts 


Energy i-^ Signals i-^ Data i-^ State vectors i-Labels i-^ 


Knowledge 
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Three-pronged approach to assist in crossing the longest yard 

• Advanced Method for visualizing evolving situations and hypotheses 

• Gaming methods for engaging analysts and personalizing interaction 

• Team of intelligent cyber agents as a virtual advisor team 
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What is a “CAVE”? 



• “Cave Automatic Virtual Environment” 

• Name inspired by "The Simile of the Cave" found in Plato's Republic 


• Public debut SIGGRAPH ’92 

• C. Cruz-Neira, D.J. Sandin, T.A. DeFanti, R.V. Kenyon and J.C. Hart, "The CAVE: Audio Visual Experience 
Automatic Virtual Environment," Communications of the ACM, Vol. 35, No. 6, June 1992, pp. 65-72. 


• CAVE™ Trademarked 

• Multiple vendors for CAVE-like systems, Immersive Projection Displays (IPDs) 
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Improved information analysis by “conservation of analyst attention”, 
enables the analyst to focus on analysis and interpretation rather than 
information searching. 











































PhNN State 


College of 

Information Sciences and Technology 


Visualization Aids 



Event Pattern Visualization 

• Located events over time, 
with time as months on the 
z axis 

• Icon shape denotes target 

• Icon color designates 
perpetrator 

• Label describes attack type 

• DTED textured with map 

• Navigation through 
environment 

• Query event for additional 
information 
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Visualization of Computer Networks 
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• Multi-layers of interpretation 

• Use of sound, colors and 
shapes to indicate data types 

• Display of temporal 
phenomena via 3rd 
dimension 
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Experiments with 

• Sonification 

• Deliberate Synesthesia 

• Negative Evidence 
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Intelligent Agents to Support Team Cognition 


Motivation and Vision 

• Effective human teams use shared 
mental models (SMM) to anticipate & 
satisfy the needs of teammates 

• Our vision: Empower software agents 
with a cognitively inspired SMM to 
support human decision makers and 
overcome information overload under 
time stress 

R-CAST Implementation 

• Inspired by Recognition-Primed 
Decision Model (RPD) 

• Integrates information seeking in a 
collaborative decision-making process 

• Supports context-centric information 
sharing 

• Enables collaborative/automated 
reasoning 



Cyber-advisory team of intelligent 
agents supports collaborative decision 

making 
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Intelligent Agents to Support Team Cognition 



Agent Architecture Features 

• Explicitly captures knowledge about the 
structure of a team (e.g., roles) 

• Explicitly capture knowledge that 
determines the teamwork process (e.g., 
team goals, team plans) 

• This knowledge enables an agent to 
dynamically anticipate needs of other 
teammates (agents or trainees). 


Knowledge Representation Schemes 

• Prolog-like knowledge based (belief) 
•MALLET: High-level language for 

representing team knowledge 

• Petri Nets: agents internal 
representation of the dynamic 
teamwork process and information 
requirements 
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Fig 1. The Composition of CAST agent 
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ELE 

Expressive Lighting Engine 


Created 2002 


an. engine LhaL aLLLomates lighting design decissionss while 
allowing arLists control the over-all express si veriest of the 
Lghling, ssuch as style and mood 


ELE is part of my disisertaLion and is used to create 
lighting in Mirage - the Interactive Greek Drama 

Advisors: Ian Horswill and Mary Poole 


• Advanced techniques for HCI lighting 

• Utilization of lighting effects to affect 

• Mood 

• Learning 

• Focus of Attention 

• Development of tools to support HCI 
creation 


Computer Gaming Techniques are 
being explored to enhance Human 
Computer Interaction and data 
understanding 



Storifi cation 


http://facultv.istPSU.edu/SeifEI-Nasr/RAEL 
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Investigation of Gaming Techniques 

• Increase productivity 

- Improved engagement 

- Enhanced focus 

- Improved engagement 

- Increased memory retention 

- Better visualization methods - lessons 
from video games 

• Increase speed of information 
extraction 

- Visual and audio semiotics 

• Enhance quality of hypothesis 

- Creating alternative hypotheses 

- Identification of contradictions 

http://facultv.ist.psu.edu/SeifEI-Nasr/RAEL 

Gaming techniques are being explored to 
improve interaction with data and 
development and evaluation of hypotheses 
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Gaming Concepts 
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Macro decision support 


GIS-like, but using game 
visualizations 

- Maps 

- Bars 

- Sprites 

To Represent 

- Events 

- Activities 

- People 

- Organizational structures 



Screenshot from Civilization IV 
[http://www.2kgames.com/civ4/home.htm] 
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Micro decision support 


• Information => 3D representation 

- story grammar (Propp 68; Schank 78) 

- Al system 

• Stereotyping 

• Stories told by characters 





Screenshot from Mirror Mask 


Screenshot from Neverwinter Nights 






















PENNStATE 


College of 

Information Sciences and Technology 


Hypothesis Creation & Evaluation 


• Use Sim-like game for 
storyboarding 

- Uncover holes 

- play with hypothesis 

• Based on story abstraction 

- Story grammar 

- Using sign of the crescent or 
others 

• Hypothesis visualization 



Scenarios Created using The Sims 2 
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Michael McNeese 


• Human Computer 
Interaction 

• Cognitive 
Psychology 

• Computer- 
Supported Work 

http://minds.ist.psu.edu 
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A 


Evaluation via a Living Laboratory 


A unique living laboratory 
environment provides the 
ability to support the 
design of effective analyst 
support tools and to 
quantify their utility 


Concept & 
Procedural Maps 


A 


Cognitive 

Engineering 

A 


Ethnography Studies 




i/11 CrisisN S' DHS 
CentersVJ"OPOFF> 


Applications and Knowledge Domains 



Application A ) ( Application B) (Application C 


2 a 


IE a 


Scenarios to Drive Team Experiments 

T 



Living Laboratory Environment 




Transactive 

Memory 


Design & Development of Support Tools 

1L. J 



TVX 



Visualization Tools Collaboration Aids Cognitive Aids 






















































PhNN State 


College of 

Information Sciences and Technology 


Application to Network Analysis 




Exploration of computer 
security analysis (cyber 
attacks and evolution) 

Decision support 

- Situation space 

- Decision space 

Explicit consideration of 
human enhanced 
analysis 
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